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SUMMARY 


Wollmer's  Mark  )v  Decision  Model  for  instructional  sequence  optimiza- 
tion was  investigated  in  a computer-assisted  instruction  (CAI)  context. 

Two  special  t’Al  programs  served  as  vehicles  for  testing  the  model;  one 
progran  (K-Laws)  taught  the  students  to  solve  DC  circuit  problems  using 
Kirchhoff's  Laws;  the  otlier  (TRIG)  gave  practice  in  manipulating  the  six 
trigonometric  ratios.  The  K-Laws  course  had  eleven  stages  or  levels; 

TRIG  had  five;  both  courses  were  arranged  in  a hierarchical  order.  The 
Wollmer  model  requires  that  transfer  would  occur  from  one  stage  to  the 
next  in  a hierarchical  learning  sequence,  and  that  these  effects  could 
be  estimated  so  as  to  produce  an  optimal  training  schedule.  To  determine 
the  effects  of  additional  practice,  half  the  calibration  sample  was  requited 
to  finish  one  successful  trial  and  half  were  required  to  have  two  successes, 
before  moving  on  to  the  next  stage.  Thirty  subjects  took  the  K-Laws  course, 
80  completed  TRIG.  Instruction  was  given  at  individual  CAI  terminals. 

All  subjects  finished  the  course,  and  learned  to  perform  satisfac- 
torily the  final  criterion  behaviors.  Practice  effects  were  unexpectedly 
slight;  people  who  had  one  success  at  each  stage  of  the  course  had  about 
the  same  criterion-problem  perform.ince  as  those  who  had  two  successes 
throughout.  The  average  time  requ.red  to  achieve  a second  success  was 
not  appreciably  different  from  that  required  for  the  first,  and  two 
successes  a^  the  immediately  preceding  level  was  no  better  than  one,  as 
far  as  transfer  to  the  next  higher  stage  was  concerned.  These  results 
indicated  that  the  Wollmer  hierarchical  model  could  not  improve  overall 
learning  mucli  by  "optimal"  scheduling  of  practice. 

One  implication  of  the  findings  is  that  in  complex  learning  hierar- 
chies where  the  top  or  most  difficult  task  consists  of  a collection  of 
previously-learned  skills,  performance  time  on  that  top  task  may  be  more 
dependent  on  the  number  of  subskills  involved  than  upon  the  number  of 
practice  trials  in  preceding  stages.  Another  implication  is  that  if 
practice  and  transfer  effects  are  to  be  significant  in  learning  this 
kind  of  hierarchically-structured  material  then  a very  large  number  of 
practice  trials  may  be  necessary. 
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ABSTRACT 


Two  CAI  programs  in  electronics  and  trigonometry  were  written  to 
test  the  Wollmer  Markov  Model  for  optimizing  hierarchical  learning; 
calibration  samples  totalling  110  students  completed  these  programs. 

Since  the  model  postulated  that  transfer  effects  would  be  a function  of 
amount  of  practice,  half  the  students  were  required  to  complete  one 
practice  problem  successfully  before  moving  to  the  next  stage;  the  other 
half  had  to  do  two  practice  problems  successfully. 

All  students  completed  the  courses  satisfactorily.  Practice  effects 
were  small;  students  who  had  one  success  in  each  stage  did  about  as  well 
as  those  who  had  two  successes.  The  Wollmor  model  was  thus  not  suitable 
for  optimizing  instruction,  in  terms  of  minimizing  overall  time,  in  these 
particular  courses.  Perhaps  the  main  reason  for  this  result  was  that, 
as  the  student  works  up  to  the  top  of  the  hierarchy,  the  sheer  number 
of  subskills  involved  in  the  finri  task  becomes  a major  determinant  of 
performance  time,  and  number  of  practice  trials  has  a relatively  minor 
effect,  unless  a very  large  number  of  practice  trials  is  given. 
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I . INTRODUCTION 


Research  at  BTL  has,  for  some  years,  been  concerned  with  different 
aspects  of  instructional  technology  required  for  more  effective  utilization 
of  computers  In  training  and  education.  Early  In  this  work  (Rlgney,  1973), 
an  outline  was  prepared  of  the  elements  that  constitute  an  Instructional 
system.  This  outline  Is  depicted  In  Figure  1.  Each  of  the  elements  shown 
there  must  be  present  In  some  form  in  a working  Instructional  system. 

The  objective  of  this  laboratory  has  been  to  allocate  Its  particular  cap- 
abilities to  research  on  appropriate  elements  in  this  diagram.  In  those 
instances  where  the  laboratory  has  produced  and  field-tested  complete 
instructional  systems,  the  best  available  elements  were  used  In  those 
parts  of  the  systems  where  the  laboratory  was  not,  at  the  time,  doing 
research. 

One  of  the  candidates  for  improvement  In  instructional  systems  is 
the  "instructional  sequence  optimizer,"  which  Is  shown  in  the  adaptive 
controller.  Atkinson  and  his  colleagues  (Atkinson  and  Paulson,  1973) 
have  convincingly  demonstrated  the  power  of  certain  types  of  optimization 
models.  The  Interest  of  this  laboratory  In  this  part  of  the  Instructional 
system  relates  to  technical  subject-matter  typical  of  technical  training 
courses  in  the  Navy.  It  was  considered  worthwhile  to  investigate  pos- 
sibilities for  developing  an  instructional  sequence  optimizer  based  on 
operations  research  techniques.  The  initial  Markov  Decision  Model 
was  described  by  Wollmer  (1973). 

Smallwood  (1962)  was  perhaps  the  first  to  propose  a definite  model; 
his  optimizer  assigned  that  lesson  segment  which  had  the  highest  utility, 
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or  highest  expected  return  in  the  criterion  score.  Estimates  of  utility 
could  be  made  from  standardization  runs  on  a sample  of  subjects.  Suppes, 
Atkinson  and  their  colleagues  at  Stanford  have  extended  optimization  of 
Instruction  into  several  dimensions;  for  instance,  Atkinson  & Paulson 
(.1973)  define  a model  which  maps  learning  performance  in  terms  of  three 
aspects:  item  difficulty,  student  ability,  and  learning  rate  parameters. 

Using  this  model,  optimization  of  a vocabulary-learning  task  was  accomp- 
lished by  estimating  the  parameters,  and  giving  practice  in  those  items 
which  promised  the  most  gain  per  practice  trial.  This  optimization  was 
very  successful,  yielding  an  efficiency  gain  on  the  order  of  40  per  cent, 
in  terms  of  time  saved.  Chant  and  Atkinson  (1973)  developed  an  optimiza- 
tion technique  for  allocating  instructional  effort  to  two  interrelated 
strands  of  learning  material.  Their  key  assumption  was  that  the  learning 
rate  for  each  of  the  two  strands  depends  solely  on  the  difference  between 
the  achievement  levels  on  the  two  strands. 

The  Wollmer  (1973)  model  assumes  that  the  course  being  taught  proceeds 
in  a definite  hierarchical  nature.  Thus  if  levels  are  numbered  consecu- 
tively with  1 being  the  most  difficult,  and  the  highest  numbered  being 
the  easiest,  mastering  of  the  material  at  a particular  level  implies 
mastering  of  the  material  at  all  higher  numbered  levels.  Furthermore,  it 
is  assumed  that  successful  completion  of  a problem  at  one  level  increases 
the  probability  of  being  able  to  successfully  complete  a problem  at  the 
next  most  difficult  level,  following  instruction  at  that  level.  This 
model  can  be  considered  a special  case  of  a partially  observable  Markov 
decision  process  over  an  infinite  planning  horizon.  Smallwood  and  Sondik 
(1973)  formulate  and  solve  such  a decision  process  over  a finite  planning 
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horizon.  The  principal  purpose  o£  tins  study  was  to  provide  data  for 
parameters  estimation  in  tliis  model. 

Researcli  in  computational  techniques  for  tlie  more  >;eneral  infinite 
horizon  Markov  decision  process  is  currently  being  dune  at  15TL  and  results 
will  be  reported  in  future  publications.  These  results  will  not  only 
offer  an  alternative  computational  technique  for  the  model  described 
above  but  also  will  allow  one  to  relax  some  of  ’ ts  more  restrictive 
assumpt ions . 
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II.  THE  TEACHING  PROGRAMS 


Two  teaching  programs  were  specially  written  as  vehicles  for 
testing  the  Wollmer  hierarchical  model.  One  of  these  programs  (K-Laws) 
taught  the  student  to  solve  DC  circuit  problems  using  Klrchhoff's  voltage 
and  current  laws;  the  other  (TRIG)  provided  instruction  and  practice  in 
manipulating  the  six  trigonometric  ratios. 

The  Kirchhoff*s  Laws  Course 

The  Klrchhoff's  and  Ohm's  relations  are  among  the  most-taught  prin- 
ciples of  science.  All  students  in  electronics  and  physics  are  supposed 
to  master  them,  and  of  course,  many  textbooks  and  courses  feature  these 
principles  throughout.  Even  so,  there  is  plenty  of  evidence  that  simple 
circuit  analysis  remains  difficult  for  many  technicians  and  students.  A 
real  problem,  apparently,  is  the  designation  or  translation  of  physical 
circuit  quantities  into  the  Kirchhoff  and  Ohm  equations.  Solving  DC  cir- 
cuit problems  via  these  equations  is  analogous  to  working  out  a "word 
problem"  in  algebra:  once  the  equations  are  set  up,  everything  can  proceed 

smoothly;  the  difficulty  is  to  translate  the  verbal  statements  can  con- 
ditions into  the  algebraic  framework. 

The  K-Laws  course  was  organized  hierarchically,  with  the  desired 
criterion  skill  at  the  end  of  the  course  being  a demonstrated  ability  to 
calculate  certain  voltage  drops  in  a three-wire  circuit  like  that  shown 
in  Fig.  2.  For  a typical  problem  near  the  end  of  the  course,  the  student 
would  be  shown  the  schematic  in  Fig.  2,  with  the  following  parameters: 
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Figure  2.  Schematic  o£  three-wire 
c ircuit 
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R draws  16  amps 
R2  draws  23  amps 
R^  draws  13  amps 
R4  draws  9 amps 

Generator  1 Is  delivering  114  volts,  polarity  as  shown 
Generator  2 is  delivering  109  volts,  polarity  as  shown 
Each  wire,  A,  B,  C,  D,  E,  F,  has  a resistance  of  0.5  ohms. 

He  would  then  be  asked,  what  Is  the  voltage  drop  across  R^? 

Such  circuit  problems  c.'.nnot  be  solved  through  guessing.  Several 
calculations  are  necessary,  along  with  careful  definition  of  the  relations 
that  prevail  in  the  circuit.  There  are  various  ways  to  find  the  desired 
answer.  For  most  technicians,  an  effective  method  Is  to  determine  the 
amount  and  directioii  of  all  the  currents,  to  convert  the  various  current 
loads  into  voltage  drops  by  multiplying  resistances  and  torrents,  and 
finally  to  set  up  Klrchoff's  voltage  law  in  one  unknown  and  solve  for 
Che  missing  voltage. 

The  requisite  skills  to  accomplish  this  final  criterion  performan^  i 
are  laid  out  in  Fig.  3,  wh»ch  displays  p presumed  "learning  hierarchy" 
for  the  college-level  subjects  that  were  used  (Gagne,  1970).  For  this 
sample  of  people,  certain  algebraic  and  verbal  skills  were  assumed;  If 
the  same  criterion  skills  had  to  be  imparted  to  seventh-graders,  then  the 
hierarchy  would  be  considerably  extended. 

There  are  two  major  paths  in  the  K-Laws  learning  hierarchy.  On  the 
left  is  what  might  be  called  the  "voltage  drop"  sequence;  here  the  student 
learns  or  reviews  the  Ohm's  law  formulas  and  practices  using  them  in 
several  circuits;  he  also  applies  the  "sign  rule"  regarding  the  direction 
of  current  flow  and  the  sign  of  the  voltage  drop  through  a resistance. 

At  the  right  side  of  the  hierarchy,  there  is  a chain  of  subskills  involving 
the  determination  of  current  direction  and  quantity  in  a three-wire,  two 
generator  circuit  with  several  passive  loads.  Here  the  information  to  be 
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Figure  3 - A Gagn^  hierarchy  for  the  K-Laws  problem  solving  procedures. 
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taught  is,  perhaps,  less  general  than  the  voltage-drop  material;  the 
solution  sequence  is  confined  to  a particular  three-wire  configuration, 
and  might  not  hold  exactly  for  similar  but  different  circuit  layouts. 

The  voltage  drop  principles,  in  contrast,  are  extremely  broad  in  applica- 
tion. Both  chains  of  the  hierarchy  have  to  be  mastered  to  solve  the 
final  criterion  problem. 

If  a learning  hierarchy  is  valid,  then  substantial  and  positive 
transfer  from  "lower"  to  "higher"  stages  should  obtain;  that  is,  those 
who  can  perform  the  subskills  "underneath"  a final  behavior  should  be 
more  likely  to  succeed  in  the  final  task,  than  those  who  cannot  accomplish 
the  subskills.  Gagne  and  his  associates  (1962)  demonstrated  such  transfer 
effects  in  a mathematics  task  with  school  children.  For  the  K-Laws  pro- 
gram, it  was  assumed  that  it  would  be  a good  instructional  strategy  to 
require  every  subject  to  demonstrate  a definite  capability  at  every  sub- 
skill level,  before  advancing  to  the  next  part  of  the  course. 

Eleven  "levels"  or  course  stages  were  defined;  those  units  corres- 
ponded roughly  to  tasks  in  the  learning  hierarchy.  Levels  were  numbered 
so  that  high  numbers  represents  easy  or  early  parts  of  the  course.  Some 
levels  were  very  elementary  and  easy,  such  as  the  lesson  involving  direction 
of  electron  flow  out  of  a battery.  The  last  two  or  three  stages,  though, 
were  quite  involved,  since  the  student  was  then  applying  several  newly- 
learned  skills  from  all  or  most  of  the  preceding  stages,  and  he  was  usually 
performing  these  skills  in  a definite  order. 

To  the  student,  a standard  teletype  terminal  was  the  major  piece  of 
teaching  hardware  in  the  set  up.  This  terminal  was,  of  course,  driven  by 
a time-shared  teaching  program,  which  was  written  in  the  BASIC  language 


and  was  stored  In  a distant  central  computer.  A random  access  slide  pro- 
jector was  used  to  display  circuit  diagrams  on  the  wall,  In  front  of  the 
student.  Two  booklets  were  also  part  of  the  teaching  package,  one  contained 
lesson  material  and  Illustrations,  the  other  had  blank  circuit  diagrams 
for  the  student  to  use  as  he  practiced  some  of  the  lessons.  A small  hand- 
held calculator  was  provided  for  calculations;  each  student  worked  a few 
problems  on  It  before  beginning  the  course. 

When  a student  appeared  for  instruction  and  first  logged  on  to  the 
teletype,  the  program  asked  him  eleven  questions  regarding  his  knowledge 
of  electronics.  These  questions  related  directly  to  the  eleven  stages  of 
K-Laws  course  content.  In  fact,  question  number  11  essentially  asked  the 
student  if  he  knew  level  11,  question  10  If  he  could  perform  the  criterion 
tasks  In  level  10,  etc.  If  he  answered  "yes"  to  a question,  then  he  was 
given  a sample  problem  to  determine  if  that  yes  answer  was  valid.  For 
example,  at  level  6,  question  6 was:  Can  you  calculate  the  resistances 

In  a parallel  DC  circuit?"  If  a student  typed  a "yes"  to  this  question, 
then  a single  test  Item  was  given  to  him,  to  see  if  he  actually  could  per- 
form. The  program  generated  a circuit  of  three  or  four  resistors  In  parallel, 
and  asked  the  student  to  figure  out  the  total  resistance  across  them.  Com- 
parison of  the  student's  answer  to  the  correct  one  was  Immediately  performed 
by  the  program  and  printed  out  for  the  student  to  see.  The  entering  skills 
test  yielded,  then,  a series  of  eleven  "yes"  or  "no"  answers,  along  with 
a pre-test  right-wrong  score  for  each  "yes"  Item. 

Once  the  student  began  the  main  K-Laws  course,  he  worked  through 
the  program  at  his  own  pace,  with  a research  staff  member  standing  nearby 
to  handle  such  matters  as  computer  shutdowns,  log-outs,  restarts,  and  timer 
resets.  The  staff  member  did  not  supplement  the  lesson  materials  or  attempt 
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to  explain  difficult  Items,  When  learning  difficulties  did  occur,  and 
the  student  was  perplexed,  he  was  told  to  go  hack  over  the  lesson  material 
carefully,  in  step-by-step  fashion.  When  a subject  finished  studying  the 
material  in  a given  teacliing  unit,  he  typed  "L»"  (for  "Done")  on  the  key- 
board, Tiie  program  then  made  up  practice  problems  for  that  teaching  unit, 
with  some  remediation  loops  automatically  keyed  to  errors. 

Generally,  the  system  worked  satisfactorily.  On  one  or  two  occasions, 
the  problem-generator  in  the  master  program  happened  to  produce  degenerate 
or  "insoluble"  problems  for  the  student.  Some  of  the  data  were  obtained 
via  time-sharing  with  a computer  just  half  a mile  across  campus;  the  rest 
of  the  data  came  by  operating  through  a time-sharing  center  some  400  miles 
away  from  the  teaching  terminal.  For  this  distant  operat ion, noisy  tele- 
phone lines  caused  total  shutdown  a few  times.  As  it  turned  out,  a few 
students  elected  not  to  use  the  slide  projector  to  display  the  circuit 
diagrams;  they  referred  solely  to  the  booklets  for  that  information. 

After  each  problem  answer  was  received  by  the  computer  terminal,  the 
system  immediately  printed  out  a "correct"  or  "incorrect"  evaluation  of 
the  answer  provided  by  the  student,  furnished  the  correct  answer,  and 
indicated  the  time  that  the  student  had  spent  on  that  problem.  If  an 
answer  was  wrong,  the  student  had  to  keep  working  in  that  same  level,  until 
either  one  correct  or  two  correct  answers  were  achieved.  Whether  the 
student  received  one  practice  problem  or  two  problems  was  decided  by  a 
coin  flip,  when  he  logged  onto  the  system. 

Thirty  subjects  completed  the  K-Laws  course.  Fifteen  of  them  were 
college  students;  fifteen  were  military  technicians  w!io  were  working  in 
electronics  or  related  fields  at  McClellan  Air  Force  Base,  California. 
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College  students  were  paid  a nominal  hourly  fee  tor  participating;  che 
military  people  were  ordered  to  appear,  and  their  only  recompense  was 
time  off  from  regular  duties.  The  subjects  differed  markedly  in  their 
familiarity  with  electronics  concepts.  Several  of  the  college'  subjects 
were  engineering  majors  and  had  completed  one  or  more  electricity  courses; 
such  students  might  claim  that  they  already  knew  much  of  the  material  in 
tlic  course,  but  no  student  could  solve  the  pretest  problems  in  the  last 
three  (most  difficult)  stages  without  some  practice  at  the  terminal.  At 
the  other  extreme  were  some  liberal-arts  majors  who  had  almost  no  tech- 
nical experience  with  voltage  drops  and  circuit  diagrams;  some  of  these 
subjects  said  that  they  "weren't  very  good  at  this  sort  of  thing,"  but 
all  of  them  persisted  and  solved  the  criterion  problems  at  the  end. 

Breaks  were  given  about  every  one -and -a -ha If  to  two  hours  during  the 
teaching.  In  most  cases  the  program  was  completed  in  a single  day  of 
six  to  eight  hours;  about  a third  of  the  subjects  had  to  appear  on  two  or 
more  days  because  of  personal  scheduling  difficulties,  system  breakdowns, 
and  the  like. 

The  Trigonometry  Course 

A short  course  in  trigonometry  (TRIG)  served  as  the  second  vehicle 
to  test  the  model.  The  TRIG  course  consisted  of  five  levels,  and  as  in 
the  K-Laws  program,  they  were  arranged  in  a strict  hierarchical  structure. 
The  levels  were  numbered  so  that  level  five  indicated  the  easiest  or  enter- 
ing lesson,  and  level  one  represented  the  most  difficult.  In  order  for 
a student  to  know  the  material  at  a given  level  (say  Level  2),  he  also  had 
to  use  significant  parts  of  the  material  at  all  higher  numbered  levels 
(say  Levels  5 through  3). 
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As  a start*  In  Level  5.  the  student  was  given  the  definition  of 


the  six  basic  trigonometric  ratios--the  sine,  cosine,  tangent,  cotangent, 
secant,  and  cosecant.  Then  he  was  presented  with  a right  triangle  which 
had  the  side  lengths  displayed,  and  was  asked  to  find  the  six  basic  ratios  for 
that  triangle.  Level  4 treated  the  cofunction  relationships;  in  this  unit 
the  student  learned  that  the  cosine,  cotangent,  and  cosecant  of  an  angle 

are  equal  to  the  sine,  tangent,  and  secant  of  the  complementary  angle. 

7 2 

Level  3 Instruction  used  the  relation  Sin  6 + Cos  0 = 1,  and  gave  practice 
In  working  out  values  from  this  equation.  Level  2 taught  how  all  the  trig 
ratios  can  be  computed  from  either  the  sine,  cosine,  secant,  or  cosecant. 
Finally,  in  Level  1.  the  moat  advanced  unit,  the  student  was  shown  how  to 
determine  all  six  ratios  from  either  the  tangent  or  cotangent.  Then  he 
was  given  either  the  tangent  or  cotangent  of  an  angle,  and  asked  to  find 
the  other  five  basic  trigonometric  ratios.  Satisfactory  performance  in 
this  last  teaching  unit  resulted  in  "graduation." 

As  in  the  K-Laws  sequence,  the  student  was  asked  questions  about 
the  material  in  a pretest  session,  before  he  began  the  instruction.  Thus, 
one  question,  keyed  to  Level  3,  asked:  "Do  you  know  how  to  compute  the 

cosine  of  an  angle  from  its  sine?"  There  were  five  such  preliminary 
questions,  one  for  each  level. 

After  the  student  answ'^red  these  five  questions  he  was  given  instruc- 
tions and  problems  at  levels  five,  four,  three,  two  and  one  in  that  order. 

A student  advanced  from  one  level  to  another  by  successfully  solving 
either  one  or  two  problems  at  that  level,  the  number  for  each  man  being 
determined  by  a random  number  generator.  In  order  for  a student  to  gain 
credit  for  a problem,  he  had  to  do  all  parts  correctly.  Thus  if  a student 
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received  a problem  at  level  five  and  gave  an  Incorrect  answer  for  one  of 
the  six  trigonometric  ratios,  he  was  Immediately  Informed  that  he  missed 
that  problem  and  then  presented  with  a new  triangle,  and  was  asked  to 
solve  for  a new  series  of  six  ratios.  Before  being  given  a new  problem 
the  student  always  had  the  option  of  reviewing  Instruction. 

Eighty  TRIG  subjects  were  run  on  a PLATO  IV  terminal.  Subjects 
were  psychology  students  who  received  "subject  pool"  course  credit  for  parti- 
cipating; each  one  was  scheduled  for  two  hours  at  the  terminal,  but  most 
did  not  require  that  long  to  finish  the  course.  TRIG  was  written  in  the 


TUTOR  language. 


III.  PARAMETER  ESTIMATION 


Two  probability  vectors  are  needed  by  the  Wollmer  (1973)  model.  The 
components  of  these  vectors  are: 


Pi 


= P 


fstudent  can  initially  solve  a problem  successfully  at 
level  1 


1 


P 

I;  - P C. 


tudent  can  perform  at  level  i-1  / student  solves  a problem 
correctly  at  level  i and  could  not  perform  successfully  at 
I level  1 before. 


A large  group  of  subjects  were  run,  as  described  in  Section  II,  to 
collect  data  that  could  be  used  for  estimating  initial  values  of  these 
parameters . 

According  to  the  model,  the  probability  that  a student  can  perform 
successfully  at  level  1 after  solving  k problems  successfully  at  level 
1+1  is  l-(l-p£)  (l"9£.fl^  * ^11  proportion  of  Incorrect  solu- 

tions at  level  i by  students  who  solved  one  problem  correctly  at  level  i+1, 
and  let  X2j  be  the  proportion  of  Incorrect  solutions  at  level  i by  students 
who  solved  two  problems  correctly  at  level  i+1.  Then  and  X2^  are 
estimators  of  quantities  as  follows: 


Xii  = (1-^i)  (l-q\^p 
^2i  ” 


(1) 

(2) 


This  is,  X^^  and  X2j^  estimate  the  probability  of  failure  at  level  i by 
students  who  solved  one  and  two  problems  correctly  respectively  at  level 
i+1.  Solving  these  for  p^  and  one  obtains: 


•Pi  = 1 - X,i/X2, 


^i+1  = 1 - "2i/^ll 


(3) 

(4) 
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The  quantities  and  X2^  may  depend  on  the  student's  answers  to 

the  pretest  questions.  Thus  in  a five-level  teaching  system  such  as  TRIG, 

A A 

if  Pj^  and  q^^j^  ate  to  he  estimated  solely  on  the  basis  of  the  answer  to 

pretest  question  j,  one  would  obtain  Xj^j^  = and 

X«j  = Y /(W. .j,  + Y.  ,)  if  the  student  answered  yes  to  pretest  question  j: 
2i  ij4^  ij46  ij+6'  / r M 

X„  • Zjj/(Xjj  + Zjj)  and  = Z^j^/  (X.  + Z^.^)  If  the  student 

answered  no  to  pretest  question  j.  (Full  details,  definitions  of  W,  X,  Y, 
and  Z,  and  data  matrices  are  given  in  Appendix  1.) 

Since  the  student  answered  either  yes  or  no  to  five  (or  eleven) 
different  pretest  questions,  there  were  five  (or  eleven)  possible  esti- 
mates of  Xjj^  and  X2^  and  consequently  of  p^^  and  Each  of  these 

estimates  for  X^  and  X^^^  were  obtained  from  the  control  group  and  are 
displayed  in  Tables  9 through  12.  There  are  several  ways  of  estimating 
the  p and  q vectors  from  this  data,  if  the  model  is  used  to  guide  students 
through  the  course.  For  example,  for  TRIG  (K-Laws)  one  might  average 
estimates  of  Xj^j^  and  X2j^  and  use  this  to  solve  for  Pj^  and  in  (3)  and 

(4).  Another  method  is  to  obtaiti  five  possible  estimates  of  p^^  and 
q^^j^  by  substituting  the  pairs  of  estimates  of  and  and  then  to 

average  these.  A third  possibility  is  to  base  the  estimates  of  Pj^ 
and  q^^]^  solely  on  the  basis  of  the  answer  to  i which  is  the  one 
specifically  directed  at  level  i.  Still  another  way  is  to  let  a yes 
answer  to  a question  be  considered  a one  score,  to  count  a no  answer  as 

A 

zero  score,  and  then  let  p^  and  be  a linear  combination  of  the  scores 

on  the  pretest  questions. 

First,  however,  it  was  necessary  to  run  initial  samples  through  the 
program  to  provide  "calibration"  data. 
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Two  samples  of  students  were  run  through  all  levels  of  each  course, 
under  either  a two  or  one  success  per-level  policy.  Allocation  of  policies 
was  by  random  assignment  to  students  at  entry  in  the  K-Laws  course,  and 
by  random  assignment  to  students  at  each  successive  level  in  the  TRIG 
course.  Answers  to  pretest  questions  were  tabulated,  but  were  not  used 
for  weights  nor  for  entry-level  decisions  (all  students  took  all  levels), 
in  these  initial,  "calibration"  samples. 
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IV. 


RESULTS:  CALIBRATION  DATA 


The  questions  of  central  importance  to  evaluating  the  usefulness 
of  this  type  of  model  are  (1)  whether  the  policy  of  requiring  two  successes 
at  a level  resulted  in  better  performance  at  the  next  level  than  the  one- 
success-at-a-level  policy,  and  (2)  whether  learning  occurred  within  a 
level,  as  indicated  by  comparing  first  success  with  second  success  data. 

The  data  in  Tables  1 and  2,  from  the  K-Laws  course,  bear  on  these  questions. 


Table  1 

Overall  Failure  Rates:  (1)  For  One  Success  Within  a 

Level  and  (2)  for  Two  Successes 
Within  a Level;  K-Laws  Course 


One  Success 

Policy  (N  = 15) 

Two  Success 

Policy  (N  = 15) 

Levels 

1 

1st 

2nd 

11 

.079 

.071 

.071 

10 

.079 

.133 

.000 

9 

.143 

.071 

.133 

8 

.133 

.278 

.235 

7 

.278 

.294 

.250 

6 

.235 

.235 

.133 

5 

.435 

.519 

.278 

4 

.308 

.315 

.167 

3 

.400 

.593 

.389 

2 

.538 

.091 

.333 

1 

.500 

.556 

.385 

Mean 

.284 

.287 

.216 

SD 

.166 

.195 

.127 

t (within  levels)  = 1.716  p(lOdf)  = .12 
t (between  levels)  = .033  p(20df)  = .97 
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Although  there  was  a slight  reduction  in  probability  of  failure  for 
the  second  success  within  a level,  from  .28  to  .22,  this  was  not  a statis- 
tically significant  difference.  This  suggests  that  the  policy  of  requiring 
two  successes  per  level  was  too  "lenient;"  that  is,  not  enough  extra 
practice  was  required  to  differentiate  between  policies.  Comparison  of 
failure  rate  means  for  the  first  success  at  a level,  column  1,  also  reveals 
practically  no  difference  (.284  and  ,287)  between  the  effects  of  the  two 
policies.  Thus,  requiring  students  to  succeed  twice  within  a level  did 
not  reduce  their  failure  rate  for  the  first  success  at  the  next  level. 

This  overall  failure  rate  was  the  same  as  that  for  students  who  were  re- 
quired to  succeed  only  once  at  a level. 

Examination  of  the  time  data  for  the  same  conditions  (Table  2), 
reveals  a similar  story. 

Table  2 

Mean  Time  (Minutes)  per  Problem:  (1)  for 
One  Success  within  a Level  and  (2)  for 
Two  Successes  within  a Level;  K-Laws  Course 


Levels 

One 

Success  Policy 
1 

Two  Success  Policy 
1st  2 nd 

11 

.80 

.97 

1.0 

10 

.88 

.86 

1.93 

9 

.93 

.97 

1.25 

1 8 

3,53 

4.00 

2.63 

1 7 

3.76 

2.16 

2,33 

6 

1.73 

3.60 

2.77 

! 5 

8.18 

7.42 

5.83 

’ 4 

8.45 

6.87 

6.07 

! 3 

7.66 

7.58 

4.88 

i 2 

4.26 

4.61 

4.83 

' 1 

1 

9.88 

7.68 

8.23 

1 Mean 

4.55 

4.25 

3.80 

SD 

3.43 

2.79 

2.31 

t 

(within  levels)  = 1.347 

p (lOdf)  = .21 

t 

(between  levels)  = 

.228 

p (20df)  = .82 

In  Table  2,  comparison  between  the  overall  means  (4.55  vs  4.25) 
for  the  first  success  within  a level,  for  the  two  policies,  obviously 
yielded  no  significant  difference,  VVhile  there  was  a slight  overall 
decrease  in  mean  time  to  achieve  the  second  success  within  a level  in 
comparison  to  mean  time  to  achieve  the  first,  this  difference  was  neither 
statistically  nor  practically  significant.  Further,  the  two-success  policy 
did  not  have  a cumulative  effect  between  levels.  Otherwise,  it  should 
have  resulted  in  a smaller  overall  mean  time  to  achieve  the  first  success 
in  a level  than  did  the  one-success  policy.  Other  data  from  the  calibra- 
tion samples  are  summarized  in  Appendix  1. 
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V.  DISCUSSION  OF  RESULTS. 


It  Is  clear  from  the  comparisons  in  Tables  1 and  2 that  different 
amounts  of  practice  on  the  same  problems  (within  a course  level)  had  only 
slight  effects  on  probability  of  failure  on  subsequent  problems  of  the 
same  kind,  and  did  not  positively  influence  performance  at  the  next  level. 
Since  the  Wollmer  Markov  decision  model  requires  that  this  positive  in- 
fluence occur,  the  model  apparently  cannot  be  applied  to  the  kind  of 
course  material  that  was  used  here.  It  also  is  clear  that  the  extra 
practice  the  student  received  under  the  two-success  policy  did  not  have 
an  appreciable  effect  on  mean  time  to  successful  solution  of  a problem 
at  the  next  level.  Again,  the  model  requires  that  the  effects  of  practice 
transfer  across  levels.  The  problems  used  in  the  K-Laws  and  TRIG  programs 
were  relatively  complicated,  in  that  each  problem  consisted  of  several 
parts,  and  required  that  the  students  perform  a series  of  operations, 
often  in  a certain  sequence.  Under  these  circumstances,  time  to  perform 
should  be  determined  by  the  number  of  operations  to  be  performed,  until 
the  practice-for-f luency  stage  is  reached, at  which  "chunking"  of  operations 
can  occur;  this  might  reduce  the  correlation  between  time  to  perform  and 
number  of  operations  to  be  performed.  If  this  phenomenon  occurs,  it  is 
likely  to  occur  only  after  long  periods  of  overlearning,  entailing  in- 
tensive practice,  indeed. 

In  a radar-intercept  trainer  (Rigney,  e^  a_l . , 1974),  students 
performed  the  same  six  mental  arithmetic  problems  over  and  over.  Over  a 
series  of  100  to  150  practice  problems,  in  10  to  20  sessions,  involving 
10  to  15  hours  of  practice,  overall  mean  latency  to  do  all  six  problems  was 
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reduced  from  68  seconds  Co  29  seconds,  or,  by  a factor  of  2.3.  It  should 
be  noted  also  that  this  was  a real-time  situation,  in  which  students  were 
driven  to  perform  faster  by  the  requirement  tc  keep  up  with  a developing 
tactical  problem.  The  present  study  did  not  impose  this  degree  of  time 
pressure  on  the  learner. 

The  different  levels  in  the  K-Laws  and  TRIG  courses  were  created  by 
introducing  new  rules,  or  procedures.  While  the  student  needed  to  use 
virtually  all  of  the  rules  or  procedures  he  had  learned  at  preceding  levels, 
it  still  seems  lik^^ly  Chat  Che  new  elements  in  problems  at  each  succeeding 
level  were  sufficiently  novel  to  reduce  inter-level  transfer  effects. 

To  see  if  a much  larger  amount  of  practice  would  affect  time  to 
perform,  three  additional  K-Laws  subjects  were  run  under  the  condition  that 
five  correct  solutions  were  required  at;  each  of  the  eleven  levels  before 
"graduation"  from  a stage  of  instruction.  This  policy  did  seem  to  promote 
more  learning  and  transfer;  the  mean  time  for  problem  solution  in  the 
final  stage  was  5.3  minutes,  compared  to  8 or  9 minutes  for  the  one-success 
and  two-success  policies.  But  a much  longer  total  training  time,  on  the 
order  of  hours,  was  required  to  get  this  two  or  three  minute  improvement 
at  the  final  level.  Because  of  this  relative  inefficiency,  the  Wollmer 
model  would  not  prescribe  such  extensive  additional  practice. 
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APPENDIX  1 


Tables  3 through  8 present  the  basic  success-failure  data  from  TRIG 
and  K-Laws  subjects.  These  data  were  cross-tabulated  so  as  to  indicate  the 
correlation  between  pretest-question  responses  and  learning  performance. 

As  an  example,  from  Table  5,  It  appears  that  there  were  seven  persons  who 
answered  "Yes"  to  pretest  question  6 and  succeeded  in  doing  their  level-6 
learning  without  a failure;  Table  7 reveals  that  there  were  only  two 
persons  who  answered  "Yes"  to  question  6,  and  then  failed  at  least  once 
when  they  actually  attempted  the  level  6 material.  As  a rough  estimate, 
then,  the  probability  is  something  like  7/9  that,  if  a person  answers  "Yes" 
to  pretest  Item  6,  he  will  go  through  the  level  6 teaching  without  a mis- 
take. Wollmer's  model  and  estimation  procedures  are  designed  to  take 
advantage  of  such  contingencies. 

Estimate  of  and  X^^  are  given  in  Tables  9 and  10  for  the  TRIG 
Instruction  and  in  Tables  11  and  12  for  the  Kirchhoff's  Laws  course. 

Tables  13  and  14  show  the  mean  times  on  time  spent  at  each  level  by 
students.  For  the  K-Laws  course,  the  computer  system  recorded  the  time 
in  minutes  with  1 second  for  less  than  one  minute,  61  seconds  for  between 
one  and  two  minutes,  and  so  forth.  To  compensate  for  this  recording 
circumstance,  29  seconds  were  added  to  all  times  shown  in  Table  14. 

Note  that  Tables  9 through  12  give  the  proportion  of  failures  for 
TRIG  and  K-Laws  based  on  the  number  of  successes  at  the  preceding  level. 
However,  at  level  5 for  TRIG  (11  for  K-Laws)  there  is  no  preceding  level. 
For  these  levels,  all  data  are  grouped  under  the  1 success  tables.  Thus 
Table  10  (12)  has  no  entry  for  level  5 (11). 


-24 


The  Number  of  Successes  by  TRIG  Students  Who  Answered 
Yes  to  the  Pretest  Question  j (W  Matrix)  and  No 
to  the  Pretest  Question  j (X  Matrix) 
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The  Number  of  Failures  by  TRIG  Students  Who  Answered 
Yes  to  the  Pretest  Question  j (Y  Matrix)  and  No 
to  the  Pretest  Question  j (Z  Matrix) 
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Pretest  Question 


Table  5 


The  Number  of  Successes  by  K-Laws  Students 
Who  Answered  Yes  to  the  Pretest  Questions 
(W  Matrix) 
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correct  solutions  at  level  i + 1 i 1 correct  solution  at  level 


The  Number  of  Failures  by  K-Laws  Students 
Who  Answered  Yes  to  the  Pretest  Questions 
(Y  Matrix) 
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Table  8 


The  Number  of  Failures  by  K-Law  Students 
Who  Answered  No  to  the  Pretest  Questions 
(Z  Matrix) 
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correct  soljtlons  at  level  1+1  1 correct  solution  at  level 


Table  9 


n of  Unsuccessful  Attempts  1>  TRIG  Course 
1 Correct  Solution  at  the  Previous  Level 
[Question  J Corresponds  to  Level  i) 
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Table  10 


Proportion  of  Unsuccessful  Attempts  in  TRIG  Course 
Following  2 Correct  Solutions  at  the  Previous  Level 
(Question  j Corresponds  to  Level  i) 
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Proportion  of  Unsuccessful  Attempts  in  Kirchhoff  Laws  Course 
Following  1 Correct  Solution  at  the  Previous  Level 
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Table  12 


1 


Proportion  of  Unsuccessful  Attempts  in  Kirchhoff  Laws  Course 
Following  2 Correct  Solutions  at  the  Previous  Level 
(In  this  Table,  Question  J is  directed  to  Level  12-i) 
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Table  13 


Time  Data  for  the  TRIG  Course  (N  = 80) 
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Table  14 

Time  Data 

for  the 

K-Laws  Course 

(N  » 30) 

Level 

Total  Time 

(Secs) 

Number  of 
Problems 

Average  Time  (Min.) 
per  Problem 
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9.91 

2 

16754 

62 

4.98 

3 

25774 

71 

6.53 

4 

1 

21755 

62 

6.33 

5 

29483 

76 

6.95 

1 

i ^ 

8455 

54 

3.10 

I 7 

6870 

60 

2.40 

8 

9031 

61 

2.95 

9 

1728 

48 

1.08 

10 

1185 

45 

.92 

11 

1244 

44 

.95 

-35- 


